CONNECTIVITY 


and CREATIVITY 
in times of CONFLICT 


Desisn perspectives for the future of work 
in Industry 5.0 environment: the digital and 
physical space in Augmented Reality uses 


Sara Muscolo?, Viktor Malakuczi? 


1Sapienza University of Rome 
sara.muscolo@uniromat.it 

2Sapienza University of Rome 

viktor.malakuczi@uniromal.it 


Abstract 

The advent of Industry 4.0 has introduced into the manufac- 
turing environment a new level of interconnection between 
machines, operations and sensors that have been able to 
automate entire processes. Along with this decentralization 
of the workforce, the new concept of Industry 5.0 aspires to 
elevate humans to the focal point of cognitive, physical and 
digital activities, by bringing advanced technologies that au- 
tonomously work beside humans in a human-centered per- 
spective. 

This is proposed to go beyond the productive purpose by 
enhancing the experience and quality of work itself, gener- 
ating questions about how design could intervene to foster 
effective communication (by limiting misunderstanding and 
conflict between the technology and the operator) especially 
in situations where digital and physical are blended. 

The object of study is the manipulation of the operator's 
work environment through the use of Augmented Reality 
(AR), investigating how it can be integrated into the work 
experience in terms of performance and psychological re- 
sponse, with particular attention to the type of interface 
placed in the worker's own spatial reality. 

After an overview of the use of eXtended Reality (XR) 
modes and how such technology can support human work, 
the focus will be placed on the category of Spatial Augment- 
ed Reality (SAR) aimed at operator training and assistance 
during production routine, which will then be analyzed from a 
design perspective involving reflections on how the discipline 
could intervene to enhance learning and use. These defini- 
tions point to raising the human factor above the task of as- 
sembly and the related operations, therefore the experience 
over the process, supporting the concept of human-centered 
manufacturing. 

From a design point of view, this becomes subject of ex- 
ploration not only regarding the configuration of the interface 
itself, but how its functionality can be manipulated to make 
the experience engaging in terms of tasks to be performed 
and human cognitive response, to arrive at considerations 
derived from a literature review with observations about 
possible ways in which the figure of the designer could act in 
such a typically engineering context. 
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Introduction 

The industrial sector is the protagonist of a current tech- 
nological evolution characterized by an increasingly close 
interconnection between digital and physical systems, re- 
sulting in a loss of the human component in relation to ever 
more autonomous and sophisticated technologies. Hence 
the advent of a new stage of Industry, the 5.0: a vision that 
intends to implement a human-centric evolution that places 
operators and their well-being at the center of the production 
process (European Commission, 2021a). 

The ways to achieve this goal will define the “work of the 
future” which, as cited in the thematic area of Cluster 4 that 
constitutes Horizon Europe's interventions, is intended to 
increase understanding of the human-machine relationship 
(European Commission, 2021b). 

This concerns in particular the coexistence of the intan- 
gible digital and tangible physical reality that ever more of- 
ten meet in working scenarios, opening new views of exper- 
imentation connecting those two worlds by using extended 
Reality (XR) as a tool: an evidence is the programme 2021- 
2022 of the Digital, Industry and Space cluster of Horizon Eu- 
rope, which comprehends “extended Reality Modelling (RIA)" 
(CORDIS, European Commission, 2022a) and “extended Re- 
ality Learning - Engage and Interact (IA)" (CORDIS, European 
Commission, 2022b). 

In this scenario, design is interpellated as a tool capable 
of elaborating new ways of relating humans and technolo- 
gies around their workspace, so that two different but col- 
laborative worlds can be merged. Indeed, if that of humans 
is unpredictable and subject to emotions, that of machines 
is predictable and automatic, but lacking human judgment, 
adaptability, and logic (Haight and Kecojevic, 2005). The goal 
of this intervention is to establish fluid communication that 
fosters and accommodates progress in favor of optimized 
processes for production and those who work in it, combin- 
ing the skills of both. 

The article takes extended Reality technology as its ob- 
ject of study, observing how its use is able to meet the goals 
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of the new industry of the future and how it promotes inclu- 
siveness and involvement of the human component. 

The exploration is articulated in a literature review focus- 
ing on the uses and limitations for which design needs to be 
involved, so that new development hypotheses can emerge 
in which the discipline is integrated into the design process 
of such technologies. 

The aim is to be able to define a vision in which future 
industry grows naturally around humans, so that they can 
evolve spontaneously in circumstances in which they are 
now alienated: in front of machines and processes whose op- 
erations do not take into account human comprehensibility 
and presence. This with particular focus then not only on pro- 
duction, but on those circumstances in which cognitive ca- 
pabilities (such as during learning and assistance) take center 
stage for the optimal performance of operations. 

In the next chapters, the dimensions touched by XR in the 
factory will be investigated, starting with new work scenarios 
in the industrial 5.0 context and its current use, and then ar- 
riving at Spatial Augmented Reality (SAR) and how it can be 
designed to optimize the course of work without devices act- 
ing as intermediaries between humans and processes. 


The evolution of industrial work and the XR 

Industry 4.0 has introduced a new level of interconnection 
between machines, operations and sensors, ensuring that 
operators have technological support at their side, designed 
however to be primarily performance-centric, opening op- 
portunities for possible human replacement in favor of au- 
tomation (Coronado et al., 2022). The 5.0 view, on the other 
hand, opposes this probability by considering the machine 
only as a tool to complement/enhance human work (without 
replacing it) while also improving its quality. 

We thus move away from the sole purpose of production 
and closer to a more experiential conception of work itself, 
which is currently instead focused on a dehumanization that 
worries not only operators but also society and governments 
(Grabowska et al., 2022). Indeed, the same authors state how 
the term “human-centric” classifies a still narrow segment of 
research that needs to become a key area of it. 

The human factor is therefore a fundamental require- 
ment in the design of Industry 5.0, which places it no longer 
as an element to be discarded but rather in a position of un- 
derstanding the technological complexity through a clear 
dialogue with it, suitable preparation, and a directional open- 
ness to development (İşcan, 2021). 

It is evident how as technologies and operations within 
the factory progress, work is destined to evolve, opening new 
possibilities for work performance and well-being, and defin- 
ing new figures and duties: the use of cognitive skills in tasks, 
is capable to increase engagement and interest while work- 
ing, making individuals more flexible. This evolution toward 
more cognitive rather than mechanical contributions is the 
result of adaptation to an increasingly digital environment, 
that makes the operator of the future able to dialogue with 
and be assisted by machines. 

The recent production methodologies open up new possi- 
bilities for increasing human capabilities: Romero et. al (2016) 
classifies various typologies in which humans are combined 
with specific technologies, describing their figure in relation 
to how they are assisted by them. Specific examples may be 
the “Augmented Operator” the result of the combination of 


the operator and Augmented Reality, or the "Virtual Opera- 
tor’ derived from the combination of the operator and Virtual 
Reality. Both types describe an operator whose skills are en- 
hanced, whether in mobility, vision, or processing, and refer 
to a type of cognitive interaction between humans and the 
mechanisms of the workspace. The latter is included in an 
industrial context characterized by Cyber-Physical Systems 
(CPS), in which information is transmitted by physical and 
virtual elements, that through feedback and data acquisition 
generate automated operational decisions. A new dimen- 
sion of human-machine interface is able to place humans 
between cyber systems and the physical world, through an 
enhancement of the human: it is the concept of Human Cy- 
ber-Physical Systems (H-CPS) elaborated by Romero et. al 
(2016) that aims precisely at a dynamic interaction with ma- 
chines in the cyber and physical worlds, enhancing human 
skills and senses through the technologies themselves. 

Indeed, industry is following the current technological 
trend toward immaterial universes, as evidenced by the re- 
port of Accenture Technology Vision 2022, which states that 
“the most ambitious companies will bring to life new physi- 
cal and digital worlds, populated by both people and artificial 
intelligences [...] gathering not only advantages over automa- 
tion, but experimenting with new forms of collaboration be- 
tween humans and machines” (Daugherty et al., 2022). This 
evidence is closely related to the use and development of 
eXtended Reality technologies, whose work is to reduce the 
distance between humans and the virtual through different 
manipulations and representations of reality, in this case in- 
tended as the work environment. 

As described by Rauschnabel et al. (2022), the main defi- 
nitions of XR, namely Augmented Reality (AR) and Virtual Re- 
ality (VR) can be easily distinguished based on the presence 
of the physical environment as part of the experience: if there 
is a local presence, it is referred to as AR. The latter can be 
declined into types such as Mixed Reality (MR), Tangible Aug- 
mented Reality (TAR), and Spatial Augmented Reality (SAR) 
that will be discussed later. 

In industrial settings, the most widely used XRs are MR 
and AR. They, once integrated, are capable of improving sev- 
eral aspects of the industrial system, in particular: 

» Time management: AR is leveraged for long and ad- 
vanced tasks in manufacturing, such as “complex set- 
ups, operations with many tasks/long cycle time and 
advanced maintenance" (Fast-Berglund et al, 2018), and 
the same authors state how connecting digital/cyber/ 
virtual and physical worlds can lead to significant time 
savings in that application area. Doolani et al. (2020) also 
call XR technologies into question, introducing new fea- 
tures such as increasing time-room flexibility. 

» Personnel training: one of the major applications of aug- 
mented reality concerns the training of operators, who 
today need to be prepared and resilient precisely as a 
response to increasingly advanced technologies. Doola- 
ni et al. (2020) define how AR can be used in this regard, 
especially for tasks that include “monitoring assembly 
line, sorting, picking, keeping, assembling, installation, 
inspection, packing, cleaning routines (process, shovel, 
sweep, clean work areas) and using hand tools, power 
tools and machinery”. Werrlich et al. (2018) testify how 
a virtual assembly phase, tested before going physical, 
can improve the training transfer. 
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» Communication with machines: scope of use can in- 
clude the communication between workers and tech- 
nologies in the industrial environment, as mentioned 
by Materna et al. (2018) regarding the programming 
of collaborative robots, in which workers were able to 
program, collaborate and readjust the cobot to new 
uses with a reasonable time. 

» Cognitive load on workers: the experiment by Hou et 
al. (2013) demonstrates how animated AR systems 
cause a positive effect of cognitive facilitation on 
workers in training, no longer having to rely solely on 
their memory to complete an assembly, and therefore 
reducing errors. 


SAR design for the workplace 

Augmented reality applications help increase human pres- 
ence in a clear dialogue with the surrounding digital and in- 
tangible environment, especially when it comes to Spatial 
Augmented Reality (SAR). In fact, Doyle Kent and Kopacek 
(2021) among the categories of intervention to foster effi- 
cient collaboration between operators and machines, include 
confidence in the use of surrounding technologies and an ex- 
ploitation of them that, however, do not make the human fig- 
ure redundant. 

This type of Augmented Reality under consideration is ca- 
pable of enhancing the user's visible scenario and influenc- 
ing the operator's involvement by offering the real and virtual 
worlds in the same view, and promoting autonomous work- 
flow by impacting on usability and cognitive workload (Rup- 
precht et al., 2021). 

Such technology moves beyond traditional eye-worn or 
hand-held displays (Figure 1), exploiting “large spatially-aligned 
optical elements, such as mirror beam combiners, transparent 
screens, or holograms, as well as video projectors” (Bimber and 
Raskar, 2005). 
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Figure 1. Principal elements for a SAR configuration. 


This typology is the most ubiquitous form of AR currently in 
use (Giunta et al, 2018), in fact it increases the actual visible 
environment, influencing, in the case of the workstation, the 
involvement of the operator, who is present between the real 
and virtual worlds, improving the performance and experi- 
ence (Uva et al., 2018). 

In fact, wearable tools for augmented reality have limita- 
tions like those listed by Siltanen and Heinonen (2020), such 
as the difficulty in seeing the real environment through glasses 


or visors, or their battery life leading to complicating their use 
when connected to external power sources. In addition, weight 
and comfort are also relevant factors for dynamic use such as 
at work. SAR on one hand offers a solution to such limitations, 
but it also has disadvantages related to the environment on 
which it operates such as visibility, surface-based distortions, 
object relationships, brightness, contrast (Kruijff et al, 2010). 

In the work environment it is supportive already from the 
design phase, Porter et al. (2010) in fact employ it as a means 
of iterative design, in which the design is projected onto the 
surface of the physical object and modified in real time, mak- 
ing the idea of the physical prototype and product much clear- 
er and more direct than from the 3D model on a computer 
screen. 

Regarding its purely industrial use, the benefits are evident 
particularly in assembly and production work tasks. In particu- 
lar, Bosch et al. (2020) state how less experienced operators 
are able to operate without or with little supervision, with in- 
creased flexibility, employment and reduced training time, 
leading to an acceptance of the technology by most workers. 

In the case of assembly, Uva et al. (2018) compared the in- 
structions viewable by Head-Mounted Display (HMD), tablet, 
paper, and projections on the work surface. The result was that 
the latter method proved to be the fastest of the assembly 
times, with greater reduction of errors and lower cognitive load, 
collecting positive testimonies especially about having hands 
free during practice. Such use has proven successful particu- 
larly in those situations where even experienced operators are 
faced with assembling products that are different from each 
other or modified (e.g, resulting from mass customization), 
being able to follow the operations without relying only on their 
memory. 

Funk et al. (2016) conducted a similar experiment, com- 
paring the same types of instructions and demonstrating how 
direct projection to the work location leads to faster assem- 
blies, fewer errors and cognitive load, while also appreciating 
the freedom in not wearing devices and having free hands. 
The increase in productivity and quality given by projected in- 
structions improves not only performance levels, but also the 
workload itself on the operator (Bosch et al., 2017). 


Applications, limitations and opportunities 
Analogous experiences related to SAR can be gathered in 
work domains different from industry, such as in medicine 
(Bin et al, 2020) or collaborative design (Ben Rajeb and Le- 
clercq, 2013) with related activities such as prototyping (Mo- 
rosi et al., 2018). Scenarios then extend by investing in well- 
ness (Mousavi Hondori et al, 2013) and culture (Ridel et al, 
2014) such as the exhibition and museum domain, where the 
authors’ past experiences show how SAR can give not only 
contextual information, but also an immersive three-dimen- 
sional view, providing an immediate and shared experience, 
as the visitor's gaze remain free while interacting. 

Despite the wide variety of uses and purposes, however, 
as of today there is no consolidated UI language that could 
help both designing and understanding the possible inter- 
actions, even though there are promising experiments, e.g. 
Schmidt et al. (2018) have performed studies on User Inter- 
face (UI) and user response to interaction. Although the SAR 
paradigm allows for immediate interaction and does not re- 
quire learning a new device, it still remains necessary to learn 
anew way of interacting with projected digital content, which 
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in 2023 does not yet have a widely shared gestural and cog- 
nitive identification. 

This is a relative disadvantage compared to HMDs that, pro- 
moted by powerful industry players, have vocabularies, 
toolkits and guidelines at their disposal, such as Microsoft's 
Design Checkpoints (Microsoft Learn, 2022) in which inter- 
action models, gestures and physical inputs, along with UX 
Elements and visual rules are well defined and distinguished. 

However, there is no common line or framework among the 
SAR case studies , making the design process of SAR experi- 
ences rather challenging. 

In the particular field of industrial SAR, effective experience 
design means not only gains of productivity, but also a facilitat- 
ed state of mind for operators. 

This can be achieved not only by optimizing the hardware 
part for greater adaptability and adoptability, but by exploiting 
the role of software often integrated into the systems, config- 
ured by the workers themselves on the factory floor. 

Such intervention should be performed and studied follow- 
ing User Experience Design concepts (Figure 2) such as those 
classified by Hillmann (2021) for XR Design: comfort and safe- 
ty, interaction (affordance, signifiers, feedback), environment 
and spatial components, sensory input (visual, audio, haptics), 
engagement (storytelling, gamification), constraints and inclu- 
sion, diversity, accessibility. 
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Figure 2. User Experience Design concepts for SAP configuration. 


Indeed, authoring tools can be useful in configuring the ac- 
tivities to be followed by the operator, aiming for optimal 
use derived from a routine that is engaging, simple and not 
monotonous. The same ease must be present for those who 
program or modify the experience itself, whether they are the 
designer, engineer, or line manager. A practical example is the 
Arkite * system that provides both the hardware workstation 
and the software setup for operators’ manual assembly guid- 
ance and training. 

While there are multiple generic purpose AR authoring tools 
(e.g. Adobe Aero, Audodesk Forge, Vuforia Studio, Areeka Stu- 
dio), for the purposes of industrial SAR these services will need 
to evolve by integrating domain-specific UX Design concepts 
in order to build an optimal process that takes into account not 
only the functioning of the various stages, but also the expe- 
rience in its use, introducing more direct feedback concepts, 
rewarding systems and gamification. 


1 The ultimate operator guidance platform. Arkite. From https://arkite.com/ 


Conclusion 

As we have seen, SAR is capable of enhancing the user's vis- 
ible scenario and influencing the operator’s experience, de- 
spite being affected by the number and type of interactions 
within the augmented space and its positioning. 

In order to ensure the optimal use of (S)AR in the industrial 
context, the presented desk research highlights the need of fu- 
ture research regarding (1) a design methodology that can be 
applied taking into account who will have to use or configure 
this technology, since “to advance the use of XR technologies 
in industrial maintenance and other similar hands-busy type of 
professional use, there is a need to develop an explicit list of UI 
and usability heuristics for XR context” (Siltanen and Heinonen, 
2020), (2) the human engagement in monotonous workflows 
such as on assembly line, through the concept of gamification 
by investigating the design of motivational elements (also em- 
ployed in the training beneficially influencing the learning pro- 
cess, as stated by Schuldt and Friedemann, 2017), (3) the ele- 
ments of interaction from human/digital inputs and outputs, 
specifically investing graphical interfaces, physical interfaces, 
supporting devices, tasks to be performed, human response 
and feedbacks received by operators while performing tasks. 

In particular, an upcoming experimental research of the au- 
thors will focus on how to reach a direct and natural gesture 
communication with projected and digital objects in order to 
develop a clearer interaction with the augmented interface, in- 
vesting visual navigation and kinesthetics studies. 

In the increasingly complex factory of the future, design 
research is fundamental for exploring and enhancing the op- 
erator’s own senses, making them meaningful and augmented 
through technologies such as SAR: these could have the power 
to increase the level of human integration in the factory, espe- 
cially in such an automated environment. 

From the perspective of the Cumulus community of world- 
wide design schools, it would be particularly interesting to ex- 
plore the differences between geographical regions and hence 
different cultures of factory work, which would influence the 
transversal applicability of new SAR solutions. On the other 
hand, design education might be influenced as well, preparing 
future designers to possible new branches of specialization, 
either by focusing on the technological medium, such as De- 
sign for XR experiences, or Design for the intelligent factory, fo- 
cusing on the industrial domain as an area for systemic design 
interventions. 
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